Data centers are facilities that store and distribute the data on the Internet, which with an estimated 100 billion plus web pages on over 100 million websites means they contain a lot of data. With almost two billion users accessing these websites, including a growing amount of high bandwidth video, the volume of data being uploaded and downloaded every second on the Internet is massive. At present the compound annual growth rate (CAGR) for global IP traffic between users is between 40% based upon Cisco's analysis (see http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/ns827/white_paper_c11-481360_ns827_Networking_Solutions_White_Paper.html) and 50% based upon the University of Minnesota's Minnesota Internet Traffic Studies (MINTS) analysis. By 2016 this user traffic is expected to exceed 100 exabytes per month, over 100,000,000 terabytes per month, or over 42,000 gigabytes per second. However, peak demand will be considerably higher with projections of over 600 million users streaming Internet high-definition video simultaneously at peak times.
All of this data will flow to and from users via data centers and accordingly between data centers and within data centers so that these IP traffic flows must be multiplied many times to establish the total IP traffic flows. Data centers are filled with tall racks of electronics surrounded by cable racks where data is typically stored on big, fast hard drives. Servers are computers that take requests and move the data using fast switches to access the right hard drives. Routers connect the servers to the Internet. At the same time as applications such as cloud computing increase computing platforms are no longer stand alone systems but homogenous interconnected computing infrastructures hosted in massive data centers known as warehouse scale computers (WSC) which provide ubiquitous interconnected platforms as a shared resource for many distributed services with requirements that are different to the traditional racks/servers of data centers.
At the same time as requiring a cost-effective yet scalable way of interconnecting data centers and WSCs internally and to each other most datacenter and WSC applications are provided free of charge such that the operators of this infrastructure are faced with the challenge of meeting exponentially increasing demands for bandwidth without dramatically increasing the cost and power of their infrastructure. Further consumers' expectations of download/upload speeds and latency in accessing content provide additional pressure.
As if these issues were not enough the required growth in volumes of data handled, reduced latency, increased speed, and reduced end-user cost are being jeopardized by the current architectural design trend for such data centers to be built in a modular manner exploiting low-cost commodity servers, rather than expensive high-end servers, see for example Barroso et al in “Web Search for a Planet: The Google Cluster Architecture” (IEEE Micro, Vol. 23, pp. 22-28) and Greenberg et al in “Towards a Next Generation Data Center Architecture: Scalability and Commoditization” (Proc. ACM Workshop on Programmable Routers for Extensible Services of Tomorrow (PRESTO08), pp. 57-62). Whilst this approach allows for tasks to be parallelized and a basic predictable performance to be delivered to users this performance is typically well below the peak performance of the servers, see for example Barroso et al in “The Case for Energy-Proportional Computing” (Computer, Vol. 40, pp. 33-37), and average approximately 20-30%.
However, whilst server and data center sizes are increasing, the power being drained by these servers and data centers is growing even faster. Whilst computational performance improvements increase approximately 200% every 2 years, energy efficiency only improves at present at approximately 100% every 2 years. Accordingly, the overall power consumption of each server is increasing at approximately 20% per annum, see for example Brill in “The Invisible Crisis in the Data Center: The Economic Meltdown of Moore's Law” (White Paper, Uptime Institute, 2007) and Humphreys et al in “The Impact of Power and Cooling on Data Center Infrastructure” (International Data Group, Market Research Document 201722, 2006). In fact, the acquisition cost of a server is now lower than the operational cost due to its energy consumption, see for example Brill and Pepeljugoski et al in “Towards Exaflop Servers and Supercomputers: The Roadmap for Lower Power and Higher Density Optical Interconnects” (Proc. 36th Eur. Conf. on Optical Communication, 2010, pp. 1-14). Koomey in “Growth in Data Center Electricity use 2005 to 2010” (Analytics Press, 2011, http://www.analyticspress.com/datacenters.html) estimated that electricity consumption in global data centers in 2010 accounted for between 1.1% and 1.5% of total electricity use globally, and between 1.7 and 2.2% for the US. At 300 TWh/year consumption and 50% generation—provisioning efficiency this represents approximately 600 TWhr of generated electricity.
Accordingly, scalability and energy efficiency have become key issues in data centers and are imposing tight constraints on the networking infrastructure connecting the numerous servers. Statistics report that about 10-20% of the equipment budget, see for example Greenberg, and about 5% of the power consumption in data centers is due to the networking infrastructure, see for example “Energy Star Program: Report to Congress on Server and Data Center Energy Efficiency” (US Environmental Protection Agency, 2007); Pelle et al in “Understanding and Abstracting Total Data Center Power” (Proc. Workshop on Energy Efficient Design, 2009); and Koomey. When taken in absolute terms, this amount of power consumed by the networking infrastructure is non-trivial representing globally the output of several tens of 1000 MW power stations and is destined to grow with the continued scaling of data centers in terms of capacity, number of servers, reduced latency, and increased access/transmissions speeds.
As such, this scalability with respect to the number of interconnected servers, as well as with the inter-server transmission data rate, and the overall power consumption are stretching the limits of today's interconnection networks based on electronics leading to optical (photonic) interconnection techniques being exploited, see for example Miller in “Device Requirements for Optical Interconnects to Silicon Chips” (Proc. IEEE, Vol. 97, pp. 1166-1185). The challenge to be addressed with any interconnection solution is to interconnect a large number of servers according to dynamically changing communication patterns, so that a large amount of bandwidth can be offered when and where required. This requires the design of high throughput and scalable architectures for the interconnection networks, with an energy consumption limited and proportional to the utilization of the network, see for example Barroso; Soteriou et al in “Exploring the Design Space of Self-Regulating Power-Aware On/Off Interconnection Networks” (IEEE Trans. Parallel Distrib. Syst., Vol. 18, pp. 393-408); and D. Abts et al in “Energy Proportional Data Center Networks” (Proc. 37th Ann. Int. Symp. Computer Architecture, 2010, pp 0.338-347).
Accordingly, the introduction of optical solutions into interconnection networks has been proposed to mitigate the issues related to electronic limitations in a similar manner as optical solutions have already mitigated limitations in high data rate long haul transmission, fanout in Fiber-to-the-Home (FTTH) applications, and are addressing evolving 40 Gb/s and 100 Gb/s point-to-point communications. Optical solutions offer the advantage of offering large bandwidth with low attenuation and crosstalk making it suitable for communications, i.e. the exchange of data packets, between servers, see for example Farrington et al in “HELIOS: A Hybrid Electrical/Optical Switch Architecture for Modular Data Centers” (Comput. Commun. Rev., Vol. 40, pp. 339-350); Cho et al in “Power Comparison between High Speed Electrical and Optical Interconnects for Interchip Communications” (J. Lightwave Technol., Vol. 22, pp. 2021-2033); Benner in “Cost-Effective Optics: Enabling the Exascale Roadmap” (17th IEEE Sym. High Performance Interconnects, 2009, pp. 133-137); Miller in “Rationale and Challenges for Optical Interconnects to Electronic Chips” (Proc. IEEE, Vol. 88, pp. 728-749); and Chen et al “Exploring the Design Space of Power-Aware Opto-Electronic Networked Systems” (Proc. Int. Sym. High-Performance Computer Architecture, 2005, pp. 120-131).
Due in part from additional flexibility wavelength division multiplexing as well as datarate across the typical link lengths within a data center interconnection networks based on photonics require that the architectural design, the selection of photonic technologies, and the operating strategies be selected and/or optimized in order to meet the requirements of power consumption, see for example Miller, Cho, Benner, Chen, and Tucker in “Green Optical Communications—Part II: Energy Limitations in Networks” (IEEE J. Sel. Top. Quantum Electron., Vol. 17, pp. 261-274) and “The Role of Optics and Electronics in High-Capacity Routers” (J. Lightwave Technol., Vol. 24, pp. 4655-4673); and scalability, see for example Farrington and Bonetto et al in “The Role of Arrayed Waveguide Gratings in Energy Efficient Optical Switching Architectures” (Optical Fiber Communications 2010, Paper OWY4), that are imposed by the current growth trend in data centers, see for example Pepeljugoski.
In many architectural designs in order to overcome the scalability limitations, multi-plane architectures have been proposed, such as those based upon space-wavelength domain architectures, see for example Gaudino in “Can Simple Optical Switching Fabrics Scale to Terabit per second Switch Capacities?” (J. Opt. Comm. Net., Vol. 1, pp. B56-B69); Raponi et al in “Two-Step Scheduling Framework for Space-Wavelength Modular Optical Interconnection Networks” (IET Commun., Vol. 14, pp. 2155-2165); and Liboiron-Ladouceur et al in “Energy-Efficient Design of a Scalable optical Multiplane Interconnection Architecture,” (IEEE J. Sel. Top. Quantum Electron., Vol. 17, pp. 377-383, hereinafter Liboiron1).
Typically, multi-plane architectures are organized based upon cards, each one with multiple ports, and fit well the modular architecture paradigm for data centers, see for example Farrington. The control of the network is delegated to a Two-Step Scheduler (TSS), see for example Raponi. The TSS addresses the problem of scheduling packet transmission by splitting the problem into two steps leading to a reduction of the problem complexity in each step, thereby leading to a reduction in the latency experienced by the incoming packets in large size networks when compared to those controlled by a single-step scheduler. Further, the TSS approach allows for the parallelization of the scheduling operations, leading to faster computation and higher scalability.
It would be beneficial to extend such multi-plane concepts to exploit space and time switching domains for the basic infrastructure with the addition of the wavelength domain to provide additional capacity to increase the throughput whilst maintaining TSS based control. It would be further beneficial for each port of the interconnection network to exploit the same electro-optic interface, leading to a simplification in implementation when compared with architectures that exploit wavelength-dependent ports, see for example Raponi and Liboiron1. Accordingly, the inventors have established space-time domain interconnection network architectures with wavelength domain overlay which overcomes prior art power consumption issues, especially at low levels of utilization, by exploiting an all-optical implementation using self-enabling semiconductor optical amplifiers (SE-SOAs). Such SE-SOAs offer the ability to act simultaneously as a switch and an amplifier, and the possibility to remain in an idle state when unused.
Within many of the architectures and implementations for optical interconnection networks, space switching plays a central role either discretely (single plane architecture) or in conjunction with time and/or wavelength switching (multi-plane architectures) such as described by Liboiron-Ladouceur et al in Liboiron1 and “A Scalable Space-Time Multi-Plane Optical Interconnection Network using Energy-Efficient Enabling Technologies” (J. of Opt. Comm. and Netw., Vol. 6, pp. A1-A11, hereinafter Liboiron2). Space switches allow multiple packets to be routed from any input ports to any output ports along different paths of the interconnection network and can be realized by exploiting optical gating elements as well as optical switching elements. An optical gating element may be controlled to either enable or block the passage of the optical packets. Previously proposed implementations of optical space switches were based on a single type of element, either a switching element such as the microring resonator, see for example Poon et al in “Cascaded Microresonator-Based Matrix Switch for Silicon On-Chip Optical Interconnection” (Proc. IEEE, Vol. 97, pp. 1216-1238) and Bianco et al in “Optical Interconnection Networks based on Microring Resonators” (Int. Conf. Comm. 2010, pp. 1-5), or a gating element such as an SOA, see for example Wonfor et al in “Large Port Count High-Speed Optical Switch Fabric for use within Datacenters” (J. Opt. Comm. and Netw., Vol. 3, pp. A32-A39) and Castoldi et al in “Energy-Efficient Switching in Optical Interconnection Networks” (Int. Conf. Transparent Opt. Netw., 2011, pp. 1-4). However, both microrings and SOAs have drawbacks. Microrings are characterized by small footprint, integrability in CMOS technology, and low power consumption they suffer from differential loss between cross and bar states and intrinsic narrowband operation. In contrast, whilst SOAs are mature, do not suffer path dependent impairments, have fast switching time, are integrable, and their inherent amplification characteristic allows operation as switch and amplifier they suffer from high power consumption.
Accordingly, it would be advantageous to exploit SOAs acting as switch and amplifier in combination with other optical elements in order to provide improved power efficient modulator-gates such that the overall power consumption of proposed heterogeneous space switches according to embodiments of the invention is reduced with respect to a space switch based solely on SOAs.
Other aspects and features of the present invention will become apparent to those ordinarily skilled in the art upon review of the following description of specific embodiments of the invention in conjunction with the accompanying figures.